Systems and methods for identifying threats and locations, systems and method for augmenting real-time displays demonstrating the threat location, and systems and methods for responding to threats

ABSTRACT

Systems for identifying threat materials such as CBRNE threats and locations are provided. The systems can include a data acquisition component configured to determine the presence of a CBRNE threat; data storage media; and processing circuitry operatively coupled to the data acquisition device and the storage media. Methods for identifying a CBRNE threat are provided. The methods can include: determining the presence of a CBRNE threat using a data acquisition component; and acquiring an image while determining the presence of the CBRNE threat. Methods for augmenting a real-time display to include the location and/or type of CBRNE threat previously identified are also provided. Methods for identifying and responding to CBRNE threats are provided as well.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-In-Part of U.S. patent applicationSer. No. 16/534,890 filed Aug. 7, 2019, entitled “Systems and Methodsfor Identifying Threats and Locations, Systems and Method for AugmentingReal-Time Displays Demonstrating the Threat Location, and Systems andMethods for Responding to Threats”, which is a Continuation-In-Part ofPCT Patent Application Serial No. PCT/US2018/017269 filed Feb. 7, 2018,entitled “Systems and Methods for Identifying Threats and Locations,Systems and Method for Augmenting Real-Time Displays Demonstrating theThreat Location, and Systems and Methods for Responding to Threats”,which claims priority to U.S. Provisional Patent Application Ser. No.62/456,007 filed Feb. 7, 2017, entitled “Personal Imaging Device forDetecting and Confirming Threat with Mapping and Augmented RecallCapability”, the entirety of each of which is incorporated by referenceherein.

TECHNICAL FIELD

The present disclosure relates to threat material detection such asCBRNE threat detection, identification, image augmenting to reflectCBRNE threats, mapping and networks to provide same.

BACKGROUND

Personnel deal with weapons of mass destruction (WMD) CBRNE andchemical, biological, radiological, nuclear and explosive (CBRNE)threats such as IED's, nerve CBRNEs, opioids, etc. throughout oursociety at almost all levels. Warfighters and first responders seeminglyare consistently put in situations wherein knowledge of the presence ofthese threats or past presence would be extraordinarily beneficial. Thepresent disclosure provides systems for identifying and mappingCBRNE-WMD or CBRNE threats and locations as well as augmenting real-timedisplays to display the location and/or type of CBRNE threat, as well asmethods for decontaminating materials and personnel that have beenidentified as exposed to CBRNE threats.

SUMMARY

Systems for identifying threat materials such as a CBRNE threat andlocations are provided. The system can include a data acquisitioncomponent configured to determine the presence of threat material; datastorage media; and processing circuitry operatively coupled to the dataacquisition device and the storage media, the processing circuitryconfigured to display real-time images depicting a location of threatmaterial determined by the data acquisition component.

Methods for identifying threat material such as CBRNE threat areprovided. The methods can include: determining the presence of a threatmaterial using a data acquisition component; acquiring an image whiledetermining the presence of the threat material; associating thelocation of the presence of the threat material with a portion of theacquired image; altering the image to depict the location of the threatmaterial; and displaying the image depicting the location of the threatmaterial.

Methods for augmenting a real-time display to include the locationand/or type of threat materials such as a CBRNE threat previouslyidentified are also provided. The methods can include: acquiring animage and storing the image on storage media; augmenting the image upondetermining the presence of threat materials; and storing the augmentedimage on the storage media.

Methods for identifying and responding to threat materials such as CBRNEthreats are provided as well. The methods can include: acquiring animage; processing the image to tag one or more features of the image toprepare a tagged image using processing circuitry; and detecting threatmaterial and associating the threat material with the tagged image usingprocessing circuitry.

A method can also include: receiving a first image of a surfaceexhibiting a colorimetric response; receiving a second image of thesurface without the colorimetric response; processing the first image bya segmentation module comprising an artificial neural network (ANN) toidentify the colorimetric response; and projecting a representation ofthe colorimetric response identified by the segmentation module on thesecond image to permit a user to identify a location of the colorimetricresponse on the surface after the colorimetric response has been removedfrom the surface.

A system can also include: an image acquisition device configured tocapture a first image of a surface exhibiting a colorimetric responseand a second image of the surface without the colorimetric response; andprocessing circuitry configured to: process the first image by asegmentation module comprising an artificial neural network (ANN) toidentify the colorimetric response, and project a representation of thecolorimetric response identified by the segmentation module on thesecond image to permit a user to identify a location of the colorimetricresponse on the surface after the colorimetric response has been removedfrom the surface.

DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Embodiments of the disclosure are described below with reference to thefollowing accompanying drawings.

FIG. 1 is a system for identifying threat materials such as CBRNEthreats according to an embodiment of the disclosure.

FIG. 2 is another system for identifying threat materials such as CBRNEthreats according to an embodiment of the disclosure.

FIG. 3 is a system for identifying threat materials such as CBRNEthreats and displaying same according to an embodiment of thedisclosure.

FIG. 4 is another system for identifying threat materials such as CBRNEthreats and displaying same according to an embodiment of thedisclosure.

FIG. 5 depicts displays of tagged images according to an embodiment ofthe disclosure.

FIG. 6 depicts displays of tagged and augmented images according to anembodiment of the disclosure.

FIG. 7 is a display of a processed image according to an embodiment ofthe disclosure.

FIG. 8 is a display of an image processing series according to anembodiment of the disclosure.

FIGS. 9A and 9B are an actual and a processed image according to anembodiment of the disclosure.

FIGS. 10A-C are actual images for processing according to an embodimentof the disclosure.

FIG. 10D is a processed image from the images of 10A-C according to anembodiment of the disclosure.

FIG. 11 is a process for processing images according to an embodiment ofthe disclosure.

FIG. 12 is a process for detecting threat materials such as a CBRNEthreat using acquired images and processing same.

FIG. 13 is a series of masked and unmasked images according to anembodiment of the disclosure.

FIG. 14 is a depiction of augmented imagery according to an embodimentof the disclosure.

FIG. 15 is a depiction of CBRNE threat detection according to anembodiment of the disclosure.

FIG. 16 is another depiction of CBRNE threat detection according to anembodiment of the disclosure.

FIG. 17 is a depiction of augmented imagery used to detect a CBRNEthreat according to an embodiment of the disclosure.

FIG. 18 is a depiction of a system for identifying threat materials suchas CBRNE threats according to an embodiment of the disclosure.

FIG. 19 is a system for identifying threat materials such as CBRNEthreats according to an embodiment of the disclosure.

FIG. 20 is a method for identifying and dealing with threat materialssuch as CBRNE threats according to an embodiment of the disclosure.

FIG. 21 is a method for confirming threat materials such as CBRNEthreats according to an embodiment of the disclosure.

FIG. 22 is a continuation of the confirmation of CBRNE threat accordingto an embodiment of the disclosure.

FIG. 23 is yet another continuation of the confirmation of CBRNE threataccording to an embodiment of the disclosure.

FIG. 24 is additional confirmation of the CBRNE threat according to anembodiment of the disclosure.

FIG. 25 is a depiction of augmented reality threat display according toan embodiment of the disclosure.

FIGS. 26A and 26B are images of capturing an image of a treated materialand the captured image.

FIGS. 27A and 27B are images of capturing an augmented live image andthe augmented live image.

FIG. 28 is another depiction of augmented reality threat displayaccording to an embodiment of the disclosure.

FIG. 29 is a depiction of augmented mapping display according to anembodiment of the disclosure.

FIG. 30 is a depiction of CBRNE threat mitigation using data fromaugmented reality display according to an embodiment of the disclosure.

FIG. 31 is an example of automated mitigation or decontaminationaccording to an embodiment of the disclosure.

FIG. 32 is an example of automated mitigation or decontaminationaccording to an embodiment of the disclosure.

FIG. 33 is at least one network that can be provided demonstratingprevious identification of CBRNE threats.

FIG. 34 is another network that can be provided displaying previouslyidentified threat materials such as CBRNE threats.

FIG. 35 illustrates a process of detecting threats using deep learningtechnology according to an embodiment of the disclosure.

FIG. 36 illustrates a block diagram of an artificial neural networkaccording to an embodiment of the disclosure.

FIG. 37 illustrates operation of a registration module and asegmentation module according to an embodiment of the disclosure.

FIG. 38 illustrates image preprocessing according to an embodiment ofthe disclosure.

FIG. 39 illustrates feature mapping processing according to anembodiment of the disclosure.

DESCRIPTION

This disclosure is submitted in furtherance of the constitutionalpurposes of the U.S. Patent Laws “to promote the progress of science anduseful arts” (Article 1, Section 8).

Multifunctional imaging devices are provided that can be configured toread the signal of threat material detectors such as CBRNE threatdetectors. The imaging device can employ cameras with multispectraloverlay to identify detection signals and/or enhance those signals to auser on a graphical display. The display may be part of a handheld orwearable device. An integrated chemical detector can provide orthogonalconfirmation of threats to reduce the incidence of false determinationsor provide semi-quantitative analysis of positive results from highlysensitive surface threat detectors. Orthogonal detection can be queuedby identification of a positive surface sensor response.

CBRNE threat detectors can include chemical agent disclosure sprays,explosive trace detectors, biological analyzers and/or collectors,bio-threat detectors and/or collectors, gas chromatography/massspectrometry instruments/detectors, mass spectrometryinstruments/detectors, ion mobility spectrometers, radiation detectors,spectroscopic radiation detectors, and/or radionuclide identificationdevices.

Embodiments of the systems can overcome poor surface detector signalvisibility for users with restricted fields of vision due to limitedlighting conditions and protective equipment. Systems can allow users torapidly scan areas where CBRNE threat detectors have been used andreceive real-time, automated information regarding threats that may bepresent. Determination of colorimetric or fluorescent signals can bemore objective by removing user interpretation of signal appearance incertain embodiments. The systems can provide threat information to usersin an augmented way that makes the threat more actionable. Byintegrating orthogonal detection capability triggered by identificationof surface sensor responses, the system can greatly reduce theproblematic incidence of false determinations through alternateconfirmation.

A contamination mapping system that overlays Agent Disclosure Spray(ADS) detection locations on a three-dimensional digital model that canbe recalled after the ADS is removed is provided. This capability cangive the warfighter greater situational awareness during decontaminationand reconnaissance operations.

Using ADS, the warfighter can identify the CBRNE via a localizedcolorimetric change, wherein red indicates the presence of the targetchemical warfare agent (CWA). There are presently formulations for G- &V-series nerve agents, sulfur mustard (HD) and opioid drug compounds,for example.

ADS is a sprayable enzymatic sensor which allows the warfighter tolocate trace CWA contamination on surfaces via a localized colorimetricchange. Another formulation that detects trace quantities of HD and onewhich detects pharmaceutically-based chemical agents is contemplated.These formulations may detect G- & V-series nerve agents and HD at tracelevels.

ADS is already being fielded for Heavy Mobile Equipment decontamination.Since 2010, warfighters have developed several Concepts of Operations(CONOPs) for the use of ADS. ADS has been used at the beginning of atriage line for vehicles used in the field to determine the presence ofCBRNE threats.

Vehicles are assessed and decontaminated according to the extent ofcontamination disclosed. ADS can then be used again at the end of theprocess to confirm that the CWA has been decontaminated properly. ADSprovides capabilities to locate contamination and focus decontaminationefforts only on local hotspots, which drastically reduces the time andlogistical burden required to perform decontamination. Further,equipment, personnel, and fixed sites can be returned to operationalcapability more quickly. ADS in these roles can act as a forcemultiplier, allowing the warfighters to maximize their limited resourcesof their labor, time, and materials to accomplish their decontaminationtasks more efficiently.

Despite existing capabilities, the warfighter requires ADS to providebetter tactical functionality at reduced life cycle costs (LCCs). Inorder to fulfill this need, improvements to the chemistry, applicationprocess, and imaging capabilities are required.

The Agent Disclosure Spray is a highly sensitive surface sensor fordetection of chemical warfare CBRNEs directly on surfaces. Once applied,the liquid formulation can provide either a yellow (negative) or red(positive) visible color response to specific chemical threats. Thepresent system has the potential for future versions to detect otherChemical, Biological, Radiological, Nuclear, and Explosive (CBRNE)threats as well as drugs or Toxic Industrial Chemicals/Materials(TICs/TIMs). Detection signals are not limited to direct visible colorbut may also include infrared (IR), ultraviolet (UV), fluorescent, orluminescent responses.

Imaging devices of the present system can provide for a system capableof using multiple cameras with and without specific optical filters tosimultaneously view an area where the surface threat detector has beenapplied and specifically identify local positive detection responses.The use of specific cameras and filters can provide for the integrationof multiple threat detectors (CBRNE, drugs, TICs/TIMs, thermal, etc.) byreading a variety of spectral signals (visible, ultraviolet, infrared,fluorescent, luminescent) in environments where visibility may belimited. The software algorithm may analyze the filtered images todetermine specifically where positive threat detection signals arepresent pixel by pixel in the field of view, for example. The signal ofinterest may then be overlaid on the visible scene image and visiblyhighlighted to the user with false color mapping on a display inreal-time. Accordingly, the device can provide actionable information tothe user. The device may be configured with built-in memory toautomatically capture, store, and recall images. Wireless features canbe included to allow for networking capability to relay information, andGPS can provide geolocation capability. Form factors for the imagingdevice can include either a handheld imager or a wearable system.

Orthogonal detection capability may also be integrated into the imagingdevice. This capability may use Ion Mobility Spectrometry (IMS), RamanSpectroscopy, or other common chemical detection methods. Byincorporating this capability, the system can significantly reduce theincidence of false positive determinations by enabling specific chemicalidentification. Some limited quantification capability is introducedbased on the sensitivities of surface detectors versus other chemicaldetectors. For instance, a positive response from both the surfacesensor and the orthogonal detector may indicate a relatively highconcentration of threat. A positive response from the highly sensitivesurface sensor but negative response from the orthogonal detectorindicates a trace concentration of threat. A negative response from bothdetectors can provide a high level of confidence that the target threatis not present. This innovation provides further novelty in that theorthogonal detector can be automatically triggered by identification ofa positive surface sensor signal.

In accordance with example implementations, the systems can include aHandheld Imaging Platform (FLIR K-series camera (Rugged and reliable(IP67) for use in firefighting, relatively inexpensive, easy to use withgloves, (3″ LCD display, multiple imaging modes, ability to saveimages); uses MSX technology for imaging (2-camera integration ofmultispectral imaging, real-time overlay of thermal and visible images,provide enhanced image resolution); translatable to visible+filteredimage overlay; Tablet-Based Imaging Platform (3D mapping of environment,real-time detection). The systems of the present disclosure can providea process for detection interpretation with an imager (1) apply threatIndicator (2) collect image (3) make determination of contamination(automated and confirmed) (4) act upon determination.

The systems and methods can accommodate filtered camera views fordetection signals, faster locating of detections, adaption for futureoperations (AR/Auto-Decon). Pixel by pixel determinations of color,clean/contaminated decisions, highlights contaminated spots to user,putting a box around adjacent contaminated pixel, can lock locationuntil user clears alert.

False-Color augmentation can be displayed using MSX technology toenhanced resolution of secondary image on visible background real-timedisplay, processing circuitry can identify threat detection signal basedon filtered image, the systems and methods can apply false color tofiltered images, enhance detection response on visible backgrounddisplay, identification tags may be provided.

The system can have hands-free imager capability, monocular device &heads-up display on mask, integrated capability for protection,disclosure visualization, and documentation.

The system can include spectral reader confirmation; after a spot isindicated as contaminated the spectral reader will confirm it is adetection and minimize user/equipment error (IMS integratedconfirmation, integrated ion mobility spectroscopy detector to confirmthreat.

The disclosure spray use can be combined with digital mapping technologyto allow the warfighter to capture and recall 3D models withsuperimposed ADS detection signals. This system can be a portable systemand may use both color and depth sensing cameras for detection signalrecognition and model generation. Simultaneous Localization And Mapping(SLAM; provided as part of a ROS (Robot Operating System), a free andopen source framework for robot software with a RTABMAP (Real-TimeAppearance-Based Mapping) capability: https://www.ros.org/) software canbe used to track the location of the sensor relative to the target as ascan is performed.

The system can include a single camera or system of cameras equippedwith area-learning capabilities configured to map a 3-dimensionalobject. The system can be optimized for scanning speed and resolution.

As part of this system, methods can be implemented which associate acolor with a specific location; for example, a red color to a specificlocation on the object, and may also include an object's track, such asa GPS map of the object's track.

The methods can be configured for detecting nerve agent simulantsparaoxon and diethyl VX on tan and green Chemical Agent ResistanceCoating (CARC), non-skid coating, and aircraft aluminum, as well asother surfaces such as SBR rubber, sand, and dark carpet, for example.

The systems can be configured to make measurements of complex 3D objectsand render models. The systems can include an outdoor sensor head andpower supply, as well as outdoor camera calibration procedures, forexample. The systems may be configured to operate in varying lightconditions and/or on varying surface types. Image processing, modelgeneration time, and mapping may be conducted of an armored fightingvehicle in 10 minutes.

The systems can provide a detailed map of contamination underoperationally relevant environmental conditions with a resolution ofrecorded contamination locations of within 3 inches or within 1 inch,when reading a contamination from up to a 15-foot standoff distance.

In accordance with at least one implementation, a 3D robotic imaging andmodeling platform can be adapted to generate a 3D model of a complexobject. These platforms can be used to generate a 3D mesh model fromseveral cameras in a fixed location to create a model; for example, aforklift carrying a steel drum as shown in FIG. 7.

The modeling system can be configured to determine the requisite numberof color and depth sensing cameras required to generate models for avehicle, and can demonstrate optimized scanning speed and resolution ona small scale. The mapping software can be customized to use sign postsfor camera position tracking. The cameras can use these sign posts torecalibrate their position if they are being moved too quickly orunsteadily during the scanning process. The system can be configured tooperate without these sign posts as well.

A color detection process can be integrated into modeling software tobegin generating contamination maps of objects. The system can beutilized to detect nerve agent pesticide simulants onoperationally-relevant surfaces such as CARC and non-skid coating. Thesystem can also be configured to provide a tag or tags that can beoverlaid with the 3D model of a bench-scale object to provide a mappedcontamination.

In accordance with other implementations, a user may visualize acontaminated area from safer distances and with higher confidence thanis possible with the naked eye or other imaging techniques. By locatingand saving information in a 3D digital map, data can be recalled afterthe ADS is removed or signals fade, focused decon and decon assuranceoperations on precise areas of concern can be obtained. Further,clearance decon may check vapor off-gassing before transportingequipment or vehicles on C130 airplanes. Additionally, data can betransmitted wirelessly to other devices or a command center to provideoptimal situational awareness among warfighters. An eventual applicationof this technology could involve integration with an automated roboticdecontamination system, further protecting the warfighter fromcontamination exposure risks.

As described herein, a digital contamination mapping softwareapplication can be developed for use with a mobile phone or tablet. Thedevice can be equipped with depth-sensing capabilities that can be usedto generate a 3D mesh network map or high-resolution model of an area.Software can link information to specific locations in these 3D maps,which can be recalled at a later time or from a remote location, see forexample FIGS. 5 and 6 demonstrating a tag (arrowed) superimposed onto anADS detection signal and remain in place when viewed from any angle ordistance.

The systems process images so that red ADS detection signals areaugmented to be more visible to the user of the device as part of thesystem. FIGS. 12-14 depict the initially developed and processed images.

The systems and methods of the present disclosure can be compatible withlive video in order to allow real-time detection signal augmentation.This algorithm will be integrated into the app to allow for ADSdetections to automatically be tagged and stored in a 3D map by theimaging device. Further capabilities beneficial to the user can bedesigned into the software, such as mapping of potential ADSinterferents and measuring distances from the user to contaminatedareas. Additionally, GPS positioning capability can be included into thesystem for reconnaissance mapping of contamination.

As indicated above, the imaging platform can create a 3D model ofobjects such as vehicles in real-time as the objects move past cameras.For example, images of objects may be captured by one or more cameras.The systems and processes of the present disclosure operate to tag aportion of the image and/or prepare an alternative image, such as analternatively colored image highlighting specifically predefined colors.The systems and processes may also create 3-dimensional images. Thealternative images can be screened for CBRNE threats and wheredetermined those threat locations tagged on the image. Referring to FIG.8, in accordance with an implementation of the systems and methods ofthe present disclosure, an object such as a fork lift carrying a load isviewed by multiple cameras. The system and methods capture the loadbeing carried on the forks, wherein, (A) RGB image of the fork liftdriving past the 3D cameras with the load on the forks boxed in by thesoftware (B) a system diagram (C) the software created 3D model of theforklift and cargo.

The systems and methods of the disclosure can recognize and dynamicallycapture an object as it is moved through a cluttered environment, whichwould be very useful in tracking moving vehicles through a fielddecontamination line.

The system can also provide for threat mapping and/or augmented recall.For example, systems can provide a new way of integrating, recalling anddisplaying information through the generation of 3D augmentedconceptualizations pinpointing the location of tagged objects in aprofiled environment. The systems can compile data from a multitude ofthreat or environmental detectors (i.e. CWA, TICS, explosives,radiological, thermal, biological, etc.), and spatially tag theinformation using 3D high-resolution mapping software (i.e. GPS, Mesh,LIDAR scanning, computer vision hardware, etc.). Sensor detection,location and time inputs can be integrated in real-time to generate arecallable augmented representation to generate a 3D projectiondisplaying the location and identity of tagged features with highaccuracy (mm-yards).

The sensor input data can also be recalled as a 3D model, versus a 3Dprojection. In this case, the 3D model displaying detector data wouldnot be overlaid in the real environment but would be viewed as a highresolution image on a tablet, smart phone, or any other viewing device.The systems can provide communication of actionable, real-timeinformation from multiple CBRNE sensors to a central data processingsystem to generate threat maps that can then be used by a centralcommand post or disseminated to end users in the field fordetector/sensor-free viewing. This information display method can createa history and high-res map of threat events that can be recalled andtransmitted on demand.

The system can improve overall situational awareness of the warfighterby allowing for the communication of CBRNE and other environmentalthreats to be viewed in real-time in the field without having the sensorthat detected the threat first hand present. Further, the system may usea multitude of position determination techniques such as LIDAR, 3D Mesh,and computer vision hardware in addition to geo location (GPS) to moreaccurately determine the location of a threat compared to that of usingGPS alone. If a threat is found on an object, the location of the threatcan also be determined in relation to the object itself versus the geolocation of the threat. This, combined with the barcoding capability ofthe innovation, can allow the user to recall the location ofcontamination even if the object that contained the contamination hasmoved. In addition, the innovation could enable the use of robotic deconby transferring locations of contaminated areas to unmanned systems, aswell as enable assessment of long term materials compatibility afterthreat exposure.

The threat mapping and augmented recall capability system and methods ofthe present disclosure can provide for a tag and scan capability thatallows an object/area to be pre-mapped and stored in a library, or amethod for scanning objects and spaces in real-time. Sensitivity andscan speed (for both the 3D model/map generation and the contaminationmap) can be tunable to the threat scenario. Rapid scans can be used toidentify gross amounts of threat contamination and slow scans can beused for trace detection. Combined superposition of chemical threat dataand thermal map may be utilized to identify potential threats in asuspect chemical lab or potential WMD environment. Innovation will beapplicable in both outdoor and indoor environments by employing GPS andlaser mapping methods. Wireless features will enable networkingcapability to relay information, and GPS can provide geolocationcapability. Form factors for the contamination map generation/viewingdevice could include commercial off the shelf 3D scanners that serve asa tablet accessory or custom cameras with custom software algorithms.Augmented reality devices can include but are not limited to wearabledevices such as glasses, heads-up displays, monocular device, hand-helddevices such as a tablet or smart phone, or spatial augmented reality,which requires a projector but no wearable or hand-held devices.

The systems and methods of the present disclosure are further describedwith reference to FIGS. 1-34. Referring first to FIG. 1, a system 10 isshown that includes a data acquisition component 12 operatively coupledto processing circuitry which is operatively coupled to storage media.While shown in line, the components of system 10 may be aligned inalternative fashions contemplated by the disclosure and recognized asoperable in the industry.

Data acquisition component 12 is a component for acquiring physicaldata. Physical temperature pressure, humidity, light, darkness,pictures, chemical, biological, are all data that can be acquired bydata acquisition component 12. In accordance with exampleimplementations, the data acquisition component can be an imageacquisition device such as a digital video camera and/or still picturecamera, for example. In accordance with other example implementations,data acquisition component 12 can be analytical devices such as CBRNEanalytical instrumentation, for example.

Data acquisition component 12 can be operably coupled to processingcircuitry 14, and processing circuitry 14 can be a personal computingsystem that includes a computer processing unit that can include one ormore microprocessors, one or more support circuits, circuits thatinclude power supplies, clocks, input/output interfaces, circuitry, andthe like. The computing system can include a memory that can includerandom access memory, read only memory, removable disc memory, flashmemory, and various combinations of these types of memory and thismemory can be storage media 16 for example. The memory can be referredto as a main memory and be part of a cache memory or buffer memory. Thememory can store various software packages and components such as anoperating system.

The computing system may also include a web server that can be of anytype of computing device adapted to distribute data and process datarequests. The web server can be configured to execute system applicationsoftware such as the reminder schedule software, databases, electronicmail, and the like. The memory of the web server can include systemapplication interfaces for interacting with users and one or more thirdparty applications. Computer systems of the present disclosure can bestandalone or work in combination with other servers and other computersystems and/or software support providers. The system is not limited toa specific operating system but may be adapted to run on multipleoperating systems such as, for example, Linux and/or Microsoft Windows.The computing system can be coupled to a server and this server can belocated on the same site as computer system or at a remote location, forexample.

In accordance with example implementations, these processes may beutilized in connection with the processing circuitry described toprovide the dynamic localized media options. The processes may usesoftware and/or hardware of the following combinations or types. Forexample, with respect to server-side languages, the circuitry may useJava, Python, PHP, .NET, Ruby, Javascript, or Dart, for example. Someother types of servers that the systems may use include Apache/PHP,.NET, Ruby, NodeJS, Java, and/or Python. Databases that may be utilizedare Oracle, MySQL, SQL, NoSQL, or SQLLite (for Mobile). Client-sidelanguages that may be used, this would be the user side languages, forexample, are ASM, C, C++, C#, Java, Objective-C, Swift,Actionscript/Adobe AIR, or Javascript/HTML5. Communications between theserver and client may be utilized using TCP/UDP Socket basedconnections, for example, as Third Party data network services that maybe used include GSM, LTE, HSPA, UMTS, CDMA, WiMax, WiFi, Cable, and DSL.The hardware platforms that may be utilized within processing circuitry70 include embedded systems such as (Raspberry PI/Arduino), (Android,iOS, Windows Mobile)—phones and/or tablets, or any embedded system usingthese operating systems, i.e., cars, watches, glasses, headphones,augmented reality wear etc., or desktops/laptops/hybrids (Mac, Windows,Linux). The architectures that may be utilized for software and hardwareinterfaces include x86 (including x86-64), or ARM.

Storage media 16 can be operably coupled to processing circuitry 14and/or data acquisition component 12. Accordingly, these three entitiesof system 10 can be included in one housing or spread across differenthousings connected via hardwire and/or wirelessly to form a network forexample.

Referring next to FIG. 2, system 20 is shown that includes an imageacquisition device 12 operably coupled to processing circuitry 24 andstorage media 26. In accordance with example implementations, imageacquisition device 12 can be a handheld device that provides digitalimagery including digital video digital imagery. In accordance withexample implementations, digital video imagery includes the capture ofmultiple still images and the alignment of same in files with each stillcaptured being acquired in time fragments. These images can be compiledin the storage media 26 for recall and play and/or manipulation byprocessing circuitry 24. Such manipulation can include marking with alocation, altering to augment the image, and/or altering to displaydepicted items as three-dimensional images.

Referring next to FIG. 3, system 30 is provided that includes an imageacquisition device 32 operatively coupled to processing circuitry 34 andstorage media 36, as well as display component 38. Display component 38can be a graphical user interface, for example, and/or it can be asimple video display, as another example. Video display 38 can beconfigured to receive real-time imagery received by the imageacquisition device and/or augmented imagery that has been processed andprovided by processing circuitry 34 and storage media 36.

Referring next to FIG. 4, an example system 40 is shown that can beconfigured with components as shown. System 40 can include imageacquisition devices 42 a and 42 b operably coupled to an electronicsboard 45. Electronics board 45 can include a combination of processingcircuitry and storage media as described herein. Electronics board 45can be operatively coupled to a display 48 as described, and uponreceiving video imagery of an object 50, replaceable bandpass filters 49a and 49 b can be placed between either one or both of cameras 42 a and42 b to provide an image to the electronics board that allows for thereal-time display to the user of an altered image that highlights thedetection of a CBRNE threat. CBRNE threats as used herein can bechemical or biological threats and in accordance with exampleimplementations, the CBRNE threat can be a nerve CBRNE that has beenidentified through ADS as described herein. In accordance with exampleimplementations, cameras 42 a and 42 b can be considered an analyticalinstrument in this regard, for they are analyzing physical data anddetermining the presence of a CBRNE threat.

As shown in system 40 multiple cameras can be utilized to acquire datarelating to a field or object. This data may all be processed to moreaccurately define the location and type of CBRNE threat.

In accordance with example implementations, images acquired utilizingthe systems described herein can have portions thereof tagged and/orportions thereof three dimensionally mapped.

FIGS. 5 and 6, provide an example tagging scenario wherein a room isscanned in 52 and portions thereof tagged, and as shown, upon theaugmentation of images captured within the room, the augmented image canbe recalled to the video display upon viewing the tagged room, whereinthe room exists at image 54 and the augmented image is shown in image56. As is shown, the augmented image displays the existence of whatcould be considered a past CBRNE threat.

Referring to FIG. 6, another image 62 is taken processed and tagged in64 and stored in storage media. When viewed in 66 from a very differentviewpoint, augmented image is displayed and displays the CBRNE threat.From another view point shown in 67 the image recalled in image depictsanother augmented image and shown to the viewer in real-time.

Referring next to FIGS. 7 and 8, imagery can be 3D mapped as shown anddescribed herein, and this 3D mapped image can be stored as an augmentedimage in the storage media and recalled at a later point. For example,with reference to FIG. 8 (A), an image can be 3D mapped when it passesthrough a set of cameras (B) to give a 3D mapped image (C). Inaccordance with example implementations, this 3D mapped image upon thedetection of a CBRNE threat, can have the location of the CBRNE threataugmented on a portion of the 3D image.

In accordance with other examples and with reference to FIGS. 9 and 10,actual and processed images are shown to demonstrate the creation of 3Dmodels. The 3D models of these images can be prepared using a stereocamera such as FLIR's Brickstream® BPC 2500 Stereo Camera(www.flir.com/brickstream; Wilsonville, Oreg.). Accordingly, depthsensing and 3D model generation can be achieved under a wider range ofreal-world environments (no interference from sunlight) than an IR-basedsmartphone for example. Referring first to FIGS. 9A and 9B as well astable 1 below, an image of a box is captured and shown in FIG. 9A. Thisimage is then processed using a stereo camera to provide a 3D image ofthe 2D box image. A comparison of the model box and generated images areshown in Table 1 below.

TABLE 1 Comparison of actual dimensions with 3D generated dimensions(FIGS. 9A and 9B) Dimension Actual Gen. 1 Gen. 2 Gen. 3 Description (cm)(cm) (cm) (cm) A -Depth 44 44 42 45 (inner margins) B-height 33 33 31 34C-width 51 51 52 50

Referring next to FIGS. 1OA-1 OD, multiple 2D images (1OA-1 OC) areprocessed to prepare a 3D image (1OD). Comparison of image actualdimensions to generated dimensions are shown in Table 2 below.

TABLE 2 Comparison of actual dimensions with 3D generated dimensions(FIGS. 10A-D) 3D Dimension Actual Generated Description (cm) (cm)A-distance 260 260 between wheels B-height 1 116 113 C-height 2 141 139D-width 119 105

In accordance with example implementations, this stereo cameraprocessing can be combined with Real-Time Appearance-Based Mapping(RTABMAP) processing available as part of Robot Operating Systems (ROS,open source https://www.ros.org/) to prepare the SLAM model generationsupplemented with color recognition to provide detection signalaugmentation capabilities with a smartphone as shown herein.

Referring next to FIG. 11, a protocol for detecting the presence of aCBRNE threat using an image acquisition device in combination withprocessing circuitry or storage media as described in the systems of 10,20, 30 and 40, is shown. In accordance with example implementations, aroom can be scanned 96 during an initialization 92 using an imagecapturing device and saving the room Area Description File (ADF) to afile 98 in storage media, for example, using processing circuitry. Frominitialization to processing in 94 can be loading the ADF room file 100.Upon loading the ADF room file, processing can commence with capturingvideo 102 and making a red image mask of the video 104 definingsegmentation threshold 106. Segmentation threshold 106 will be describedwith reference to FIGS. 13-17, for example, and then determining thepresence or absence of valid segments, where valid segments are notdetermined, a return to capturing video 102, where they are determined,proceed to apply object filters 110 and then generating alternativeaugmented reality tag 112 and saving tag with room file 114.

In accordance with example implementations, a more detailed processingdepiction is shown that includes images with representation of FIG. 12that is consistent with the processing step 94 of FIG. 11. As can beseen in FIG. 12, the input frame can be in RGB-HSV-YUV-YCbCr-Lab etc.The initial processing step can include transforming a 3-channel colorimage into a monochrome normalized red image matrix with the valueswithin the matrix being represented by values between 0 and 1 (orbetween 0 and 255 to visualize the values). In certain cases, the morered a certain pixel has, the closer to 1 the value will be, and if thereis no red color at all, it should have a value of 0. This can form asignal rather than noise determination as is used in typical detection.The adaptive segmentation threshold is then defined so that the matrixcan be segmented into objects that can be referred to as blobs. Withthis threshold in mind, the image can be constructed into objects orblobs i.e., with red pixels with clusters into blobs. Then certainfilters are applied on the obtained blobs. The color filters such asbrightness, redness, size filters, shape filters can be applied. In thefinal step, an output image is generated that includes a list ofobtained objects or regions that are colored red, and those objects canbe monitored over time to see changes.

Referring next to FIGS. 13-14, more depictions of the conversion ofstored imagery into segmented imagery and then finally the formation ofblobs are shown. For example, Clear UV-ADS; Additive UV-ADS; and/orClear NI R-ADS samples were prepared and processed. As can be seen, theprocessed images provided a much clearer indication of the agent threat.

As can be seen in FIG. 14, sections of objects were treated, and theimagery processed to shown multiple color depictions during imagemasking and segmentation to form blobs.

Referring particularly to FIG. 15, the image device 130 can be used todetect the presence of a CBRNE threat through segmentation of imagescaptured by the device. In accordance with example implementations,device 130 can be coupled to or the data associated with anotherdetection method. In this case, for example, ion mobility spectrometer134 data can be acquired and processed to give an additional level ofdetection that can be combined with the data received by apparatus 130.

As can be seen in more detail in FIGS. 16 and 17, the color difference(shown as 132 in FIG. 15) is graphed showing the color differencebetween the visible ADS white image and the monochrome camera maskedimage. This difference can be considered the difference betweendetecting or not detecting a CBRNE threat.

Referring next to FIG. 18, in accordance with an example implementation,system 160 is shown. System 160 can include data acquisition component162 operatively coupled to processing circuitry 164 and storage media166, which is operatively coupled to display 168. In accordance withexample implementations, data acquisition component 162 can include bothan analytical instrument 170 and image acquisition device 172. Inaccordance with example implementations, the image acquisition device isas described herein, and the analytical instrument can be as describedherein as well, such as, for example, CBRNE detector networking mappinginstrumentation. In accordance with example implementations, analyticalinstrument 170 and image acquisition device 172 as well as display 168and processing circuitry 164 and storage media 166 can all be part ofthe same device. In accordance with alternative embodiments, thesecomponents can be part of separate devices that are connected wirelesslyor through a network, for example.

Referring next to FIG. 19, in a more detailed view, personnel 180 can beequipped with one or more of analytical instrumentation 170 which may beused to acquire data and display same at 168 using, for example,heads-up displays and/or augmented reality displays, and these displayscan include the displays shown in 182. As described herein, thesedisplays can be augmented to demonstrate the location of a CBRNE threat.The displays and instrumentation can be coupled to processing circuitryand/or storage media 164/166. As can be seen, displays 182 can beaugmented to display CBRNE threats and/or 3-dimensional imagery forexample.

Referring next to FIG. 20, a device 185 can have an additionalanalytical device 186 such as an ion mobility spectrometer integratedtherein that may be coupled to a reading device, that may be removablycoupled to device 185 and processed using separate processing circuitry187. In accordance with example implementations, data 188 can be takenfrom an image 189 to identify a CBRN E threat, for example, utilizingthe methods and processes described herein.

Referring next to FIG. 21, a system 190 is shown, and this system caninclude the method of detecting, confirming, documenting, recalling, andacting upon a CBRNE threat. The detection has been described herein, andthe confirmation has been through orthogonal confirmation as describedherein with both video display as well as additional physical data suchas CBRNE threat detection via ion mass spectrometry, for example. TheCBRNE threat can be documented and stored as described herein, whereinthe CBRNE threat is part of a tagged image and/or an augmented image,and then the CBRNE threat can be recalled via a video heads-up displayto show the operator the presence of the CBRNE threat when viewingreal-time imagery. In accordance with example implementations and withreference to confirmation, FIGS. 22-23 are provided.

Referring to FIG. 22, a room can be entered at 202 by an operatorhandling a device that has one of the systems described herein. Aninitial scan of the room can be taken at 204, and then utilizing theprocessing circuitry, a 3D mesh can be generated of the background andstored in storage 206. This 3D mesh is not seen by the user.

Referring next to FIG. 23, the methods of 90 in FIG. 11 and FIG. 12 canbe applied to provide a box 209 around a CBRNE threat area in view 210,and this box can be viewed as an augmented image in real-time. Referringnext to FIG. 24, as shown in views 212, 214, 216 and 218, the box 209 istagged and remains there as an augmented image upon real-time display,even when viewed from other angles, as shown in the multiple angles of212, 214, 216, and 218. Referring next to FIG. 25, the location 209 ofthe box is bound not only to the image, but to the 3D mesh that wascreated of the image, and this can allow users to see wherecontamination was, even after cleanup or signal fading.

In accordance with yet another example and with reference to FIGS.26-27, color detection and SLAM can be performed using a smartphone(Lenovo Phab 2 Pro) having area learning capability. With reference toFIGS. 26A and 26B capturing images is shown and the captured image of amaterial exposed to a disclosure spray. Accordingly, device 230 can beused to prepare an image of material 232 treated with ADS and disclosingdetected material 234. In this case detection of a red positive signal234 is initiated by pressing the Search button on the right of thedevice screen.

Referring to FIGS. 27A and 27B, capturing augmented images and theaugmented image is shown with augmentation 236. This augmentation usingSLAM can be seen over live video and remains associated with thedetected location with movement of the material outside and returning tocamera view as well as when the camera is aimed away from the materialand then back again. Augmentation 236 will also stay in place afterremoval of detected material 234. This can provide true confidence whendetecting and removing the very dangerous agents detected using ADS asthe using can return to the actual location of detection after theindicator has been removed with the detected material.

Referring next to FIGS. 28-31, multiple examples of the augmentedreality threat display are depicted, and this augmented threat displaycan be facilitated through a smartphone app, for example, and the threatlocation can be pinned by GPS location, thereby giving it anotherassociation with a location as described herein. It is contemplated thatthe display can be a graphical user interface, which would allow for theuser to tap on the symbol to provide range, bearing, and status of thethreat as well. Referring to FIG. 29, an overall system can provide anevent notification to a network and an app platform can be apprised ofthis notification and augmented reality imaging can be used to view thethreat in real-time. Referring to FIG. 30, in an additional application,IEDs can be mapped utilizing these systems by pulling in data frommultiple network users and then transferring this data to smartphoneapps and providing augmented reality to those viewing the area throughdisplays that are on the network system.

Referring to FIG. 31, the augmented reality can be recalled for CBRNEthreat mitigation. In accordance with an example implementation, awarfighter can be using a video display that can be a part of thesystems and methods of the present disclosure that can recall augmentedreality that is related to a tagged vehicle, for example, and thistagged vehicle may have a CBRNE threat that has been detected, and uponviewing this tagged vehicle, the augmented reality will display a box ornotify of a CBRNE threat, for example. This data would be provided toall users connected to the system and allow the user to seecontamination in real-time, even after signal fades at any angle, andany lighting condition.

Mitigation can take place, for example, with reference to FIG. 32. Inaccordance with example implementations, items that may be exposed toCBRNE threats routinely undergo decontamination. These items can includevehicles, for example, such as armored vehicles and/or planes.Mitigation typically takes place present day by an operator cleaning thevehicle in a tented area. The present disclosure provides for theautomation of this mitigation, wherein image acquisition devices andprocessing circuitry and storage media are utilized to both determinethe presence of a CBRNE threat and manipulate robotic or automaticcleaning systems to remove the CBRNE threat. For example, once a CBRNEthreat is recognized on a Humvee, the Humvee is tagged, 3D mapped, andthe CBRNE threat marked on the Humvee. The Humvee can then proceed to atented area or enclosed area with an automated or robotic cleaningsystem and the CBRNE threat removed by the system directing theautomated cleaning system to the CBRNE threat.

Additionally, with reference to FIGS. 33 and 34, a network can beprovided wherein a historical library of CBRNE threats as well as IEDsor any kind of threat is mapped. Access to the network can be obtained,and individuals allowed to access the network using video displays,augmented reality, or other parts of data systems. These can be usefulfor the warfighter and/or the first responder, for example.

In addition to the applications mentioned above, an additionalapplication of the innovation includes being used as aclearance/verification tool and recall previously contaminated vehicles,rooms (enter/exit clearance), and personal protective equipment toprovide contamination location and decontamination assurance.

As discussed, data from threat or environmental detectors may bedisplayed to a user in an augmented reality view that gives a preciselocation of the threat in a profiled environment. While the abovediscussion provides for integration of the detector data with a 3D modelgenerated with a scanning device, additional embodiments are alsoprovided.

For example, in some embodiments, deep learning technology may be usedto register the dimensions and features of a contaminated environment,segment the contamination data, and then reorient the contamination dataso the augmented view of the user is accurate from any perspective. Deeplearning can also be used to identify objects in a scene, such as avehicle type, and superimpose a pre-generated CAD model of that objectover the user's view. Detector data may be associated with thecoordinates on the model that correspond with the actual environment. Assimilarly discussed above, the contamination data processed andidentified by such deep learning techniques may be displayed on atablet, smartphone, and/or through a heads up display. Data can berecalled at a later time when the sensor is not present, and saved ortransmitted to multiple end users.

Such techniques may be used to improve the overall situational awarenessof the warfighter by enabling the precise location CBRNE and otherenvironmental threats to be captured and viewed in real-time, or to beviewed at a later time in the field without having the sensor thatinitially detected the threat first hand present. In particular, anentire 3D model of the interrogated object or area need not be generatedin all cases (e.g., the interrogated environment may be determinedthrough deep learning processing of images captured of the environment).This negation of post-processing to render 3D digital information cansignificantly reduce time necessary to render data and display it tousers, and also reduces hardware requirements for operation. Deepleaning processing may be employed using relatively inexpensive machinevision cameras that don't require precise calibration or extensiveexpertise to operate.

In various embodiments, images are captured of one or more objects in anenvironment before, during, and after application of the ADS on asurface under review (e.g., a surface of an object, vehicle, person,and/or other potentially contaminated surface). As discussed, the ADSmay exhibit a localized colorimetric change in the presence of acontaminant. Deep learning technology may be applied to the capturedimages to identify the detected areas exhibiting the positivecolorimetric change through a segmentation process. Deep learningtechnology may also be applied to the captured images to perform imageregistration among the various captured images in order to geometricallyalign the images in one or more coordinate systems. In variousembodiments, one or more artificial neural networks (ANNs) may betrained to prepare and refine the operation of the segmentation andimage registration processes.

The various images with and without the ADS present may be processed inaccordance with various techniques to detect features and dimensions inthe images. After the ADS is removed (e.g., thus removing the presentlyvisible positive colorimetric change), due to the digital capture of thepositive color and fusion with the registered location, an augmentedreality display of the exact locations of the colorimetric indicationsmay be provided. For example, in some embodiments, a pixel-level featurematching correlation process may be performed on images with and withoutthe ADS present to accurately place the augmentation on the user's viewregardless of orientation to the object or surface.

As described herein, threat material identification, tracking, andmitigation may be performed by a system including imaging devicesconfigured to capture imagery of a contaminated environment (e.g.,objects and/or associated scenery and contaminant indicators on surfacesof such objects/scenery), image processing modules configured toidentify the extent of the contamination on the objects/scenery (e.g.,via various contamination contours and/or associated object/scenerycontours/models), and augmented reality displays (e.g., goggles, tabletdisplays, phone displays, and/or other handheld or worn portabledisplays with integrated cameras, orientation sensors, and/or positionsensors, as described herein).

For example, in an embodiment employing ADS, a contaminated vehicle maybe driven into view of a camera configured to capture imagery in thevisible, infrared, ultraviolent, and/or other spectrum or combination ofspectrums associated with the ADS, particular types of contaminants,and/or object features and/or tagging of the contaminated vehicle. Thecamera may capture one or more images of the contaminated vehicle, suchas from a single vantage point or from multiple vantage points (e.g.,with each image being stored or linked with corresponding orientationand/or position data—also referred to as pose data), sufficient indetail to identify the type of vehicle and/or various features and/ordimensions of the vehicle and/or the surrounding scenery (e.g.,including one or more pose standards—3D dimension standards that can beplaced on the ground near the vehicle to link image poses to a commoncoordinate frame).

After the pre-contaminant-identification images are captured, thevehicle may be sprayed with ADS, and the imaging of the vehicle andscenery may be repeated (e.g., to generate contaminant-identificationimages) to capture various characteristics of the ADS, such as colorchanges of the ADS (e.g., a colorimetric change indicating the presenceof a particular contaminant), and/or the temporal evolution of suchcolor changes (e.g., corresponding to the concentration of a particularcontaminant and/or the type of contaminant). Such imagery may beprocessed to identify the type, concentration, extent, and/or othercharacteristic of the contaminant, for example, and to characterize theextent of the contamination on the vehicle, such as relative to apre-generated 3D model, captured using a point cloud-generatingtechnology, such as LIDAR or stereocameras, of the vehicle generatedfrom the pre-contaminant-identification images, for instance, orretrieved from a library of pre-generated models for the identified typeof vehicle or the specific vehicle being imaged. Such contaminantcharacteristics may be recalled and projected onto a live video feed ofthe vehicle from the perspective of a portable display device, such asan augmented reality display device, even after the threat disclosurespray has faded or washed away, and even after the vehicle has beenmoved.

In some embodiments, the contaminant-identification images and ADScharacteristics may be used to direct decontamination procedures (e.g.,washing, replacing vehicle elements, and/or other decontaminationprocedures), and the contaminant characteristics (e.g., particularlycontaminant contours associated with the extent of the contamination onthe exterior of the vehicle) may be used to generate augmented realityviews of the precise location of contamination in order to track theefficacy of the decontamination efforts (e.g., including reapplicationof the threat disclosure spray to previously-identified contaminatedareas of the vehicle after washing to ensure that no residualcontamination is present). Importantly, such decontamination trackingmay be performed without regenerating a 3D model of the vehicle andinstead use the previously generated 3D model and real-time featureidentification to reorient and scale the contamination contours to matchthe extents of the contamination as presented in thecontaminant-identification images. As a result, embodiments may speedcontamination identification, tracking, and mitigation relative to othersystems requiring relatively slow and/or expensive or complexcoordination of 3D scanners and model generators, for example, or evenmultispectral or relatively high resolution imaging devices, asdescribed herein.

Turning now again to the drawings, FIG. 35 illustrates a process 3500 ofdetecting threats using deep learning technology in accordance with anembodiment of the disclosure. In various embodiments, process 3500 maybe performed by any of the processing circuitry, image acquisitiondevices, displays, storage media, and/or other components disclosedherein as appropriate. Although process 3500 will be discussed primarilyin relation to colorimetric changes associated with ADS, other types ofchanges (e.g., fluorescent, luminescent, spectroscopic, and/or others)may be detected as appropriate (e.g., in response to other types ofinterrogating sensors (e.g., materials) besides ADS that may be used tointerrogate to a surface of interest to interact and detect with one ormore threats disposed on the surface).

In step 3510, an artificial neural network (ANN) is trained to detectthe presence of a colorimetric change associated with ADS in accordancewith deep learning techniques. For example, FIG. 36 illustrates a blockdiagram of an ANN 3600 according to an embodiment of the disclosure. Insome embodiments, ANN 300 may be implemented by any of the variousprocessing circuitry disclosed herein. In some embodiments, ANN 3600 maybe implemented in accordance with a U-net convolutional neural networkarchitecture.

As shown, ANN 3600 includes various nodes 3602 arranged in multiplelayers including an input layer 3604 receiving one or more inputs 3610(e.g., any of the various images disclosed herein), hidden layers 3606,and an output layer 3608 providing one or more outputs 3620 (e.g.,segmentation results). Although particular numbers of nodes 3602 andlayers 3604, 3606, and 3608 are shown, any desired number of suchfeatures may be provided in various embodiments.

In some embodiments, ANN 3600 may be used to detect the presence of acolorimetric change (e.g., a colorimetric response) associated with ADSon a surface under review in one or more images. In this regard, ANN3600 may operate on images where ADS is present and segment portions ofthe image where a colorimetric change is present (e.g., identify,extract, and/or isolate a portion of the image corresponding to thecolorimetric change). The segmentation results may be provided by ANN3600 at outputs 320 where they are further processed in accordance withvarious techniques disclosed herein.

Accordingly, in step 3510, ANN 3600 may receive sets of images with ADSexhibiting a colorimetric change (e.g., a red response in someembodiments) on various surfaces. Also in step 3510, ANN 3600 mayreceive sets of images of various surfaces without ADS present butexhibiting similar colorimetric values (e.g., random red objects taken,for example, from a Common Objects in Context (COCO) dataset in someembodiments).

In various embodiments, the training performed in step 3510 may beperformed on an iterative basis to train and refine the response of ANN3600 to ADS colorimetric changes. As a result of such training, ANN 3600may be available to detect and ADS colorimetric changes in imagescaptured during process 3500 as further disclosed herein.

In step 3515, one or more image acquisition devices are operated tocapture one or more images (e.g., still images and/or video imagescorresponding to visible light, infrared, thermal, and/or otherwavelengths) of a surface of interest (e.g., a vehicle, person, and/orother surface under interrogation) before ADS has been applied to thesurface. For example, step 3515 may include positioning the surface inthe field of view of one or more image acquisition devices and capturingone or more images of the surface. The various images discussed withregard to process 3500 may be stored, for example, in any of the variousstorage media disclosed herein.

In step 3520, ADS is applied to the surface of interest as discussed. Instep 3525, ADS will exhibit a colorimetric change if a detectable threatis present (e.g. a CBRNE threat in some embodiments). In this regard, insome embodiments, ADS will change to a red positive color at thelocation of a detectable threat on the surface of interest (e.g., ADSmay chemically interact with threat material disposed on the surface).

In step 3530, one or more image acquisition devices are operated tocapture one or more images of the surface of interest after thecolorimetric change is exhibited. For example, if a threat is present,the images captured during step 3530 may appear similar to the imagescaptured during step 3515, but with particular areas of discoloration(e.g., red portions) associated with colorimetric changes resulting fromthe application of ADS (e.g., or any other appropriate type ofinterrogating material) and its interaction with one or more threatmaterials on the surface.

In step 3535, ADS is removed from the surface of interest. In step 3540,one or more image acquisition devices are operated to capture one ormore images of the surface of interest after ADS removal. In thisregard, it will be appreciated that the localized colorimetric changewill no longer be visible on the surface of interest in the imagescaptured during step 3540.

Thus, it will be appreciated that following step 3540, a variety ofdifferent images will have been captured of the surface of interest.Specifically, these include images captured before ADS is applied (step3515), after ADS is applied (step 3530), and after ADS is removed (step3540). In various embodiments, one or more of these images may beprovided to ANN 3600 and/or other processing circuitry to perform imageprocessing as disclosed herein.

For example, FIG. 37 illustrates various input images 3702, 3704, and3706 in accordance with an embodiment of the disclosure. In this regard,image 3702 is captured during step 3515 with a surface of interest 3701present. Image 3704 is captured during step 3530 after ADS has beenapplied resulting in a colorimetric change 3703 (e.g., also referred toas a colorimetric response) on surface 3701. Image 3706 is capturedduring step 3540 after ADS has been removed resulting in the absence ofthe colorimetric change 3703 on surface 3701.

In step 3543, one or more of images 3702, 3704, and 3706 may undergopreprocessing by appropriate processing circuitry to prepare the imagesfor further processing by registration module 3710 and segmentationmodule 3720. A variety of preprocessing techniques are contemplated andmay be used in various embodiments including, for example, adjustingimage size, filtering (e.g., median blur, Gaussian blur, and/or otherfiltering), edge detection (e.g., Laplacian, Canny, and/or other edgedetection techniques), adjusting color space (e.g., BGR, HSV, grayscale,and/or other color space adjustments), adjusting contrast (e.g., linear,gamma, and/or other contrast adjustments), adjusting brightness, and/orother preprocessing as appropriate. In some embodiments, performing thepreprocessing of step 3543 may permit registration module 3710 toprovide improved feature detection and matching and may permitsegmentation module 3720 to provide improved segmentation processing.

FIG. 38 illustrates examples of preprocessing performed during step3543. As shown, an image 3804 captured during step 3530 is provided andincludes colorimetric changes 3803 on a surface of interest 3801 (e.g.,similar to image 3704 of FIG. 37 as discussed). In one embodiment, ablurring filter may be applied to image 3804 to provide a preprocessedimage 3804A. In another embodiment, the brightness and contrast of image3804 may be adjusted to provide a preprocessed image 3804B.

In step 3545, one or more of images 3702, 3704, and 3706 (and/orpreprocessed versions thereof) are received and processed by aregistration module 3710 as shown in FIG. 37. In various embodiments,registration module 3710 may be implemented by any of the variousprocess circuitry disclosed herein. In some embodiments, registrationmodule 3710 may operate in accordance with deep learning principles(e.g., using an ANN in some embodiments).

In some embodiments, registration module 3710 may process images 3702,3704, and 3706 to align them with each other using one or moretransformation processes (e.g., to transform the images into a commoncoordinate system). In this regard, it will be appreciated that images3702, 3704, and 3706 may be captured from different relativeorientations between the image acquisition devices and surface 3701(e.g., particularly in the case of handheld image acquisition devicesand/or if surface 3701 is in motion). Thus, registration module 3710 mayapply one or more transformation processes to images 3702, 3704, and3706 such that they may be referenced by a common coordinate system.

Various registration processes may be used such as, but not limited to:Oriented FAST and Rotated BRIEF (ORB), Brute-Force matcher, ScaleInvariant Feature Transform (SIFT), AKAZE, Enhanced CorrelationCoefficient (ECC), Simple Elastix, and/or others as appropriate. In someembodiments, registration module 3710 may perform ORB processing onimages 3702, 3704, and 3706 to detect features and dimensions of surface3701 and colorimetric change 3703. In some embodiments, registrationmodule 3710 may further perform a Brute-Force matcher process on theresults of the ORB processing to perform pixel-level feature matchingcorrelation between the various images 3702, 3704, and 3706 toaccurately place an augmented representation 3705 of the colorimetricchange 3703 on the images for viewing by a user.

FIG. 39 illustrates an example of feature mapping provided by ORBprocessing during step 3545 to perform image registration. As shown, animage 3904 captured during step 3530 is provided and includescolorimetric changes 3903 (e.g., similar to image 3704 of FIG. 37 asdiscussed). An image 3906 captured during step 3540 is also providedafter removal of the ADS and thus without colorimetric changes 3903(e.g., similar to image 3706 of FIG. 37 as discussed). As a result ofprocessing images 3904 and 3906 by registration module 3710 during step3545, a plurality of features are matched (e.g., correlated) betweenimages 3904 and 3906 as denoted by lines 3944. It will be appreciatedthat lines 3944 are provided for purposes of illustrating the results offeature mapping and do not form part of the actual images 3904 and 3906.

In step 3550, one or more of images 3702, 3704, and 3706 (and/orpreprocessed versions thereof) are received and processed by asegmentation module 3720 as shown in FIG. 37. In various embodiments,segmentation module 3720 may be implemented by any of the variousprocess circuitry disclosed herein. For example, in some embodiments,segmentation module 3720 may be implemented by ANN 3600 as discussed. Inthis regard, segmentation module 3720 may detect colorimetric change3703 in image 3704 and isolate portions of image 3704 where colorimetricchange 3703 is present. As also shown in FIG. 37 and denoted by arrow3712, segmentation module 3720 may also operate on images that have beenprocessed by registration module 3710. As a result, the colorimetricchange 3703 detected by segmentation module 3720 may be aligned ontovarious images to provide a projected representation of the colorimetricchange 3703 as further discussed.

In step 3555, processing circuitry projects (e.g., superimposes) arepresentation of the detected colorimetric change 3703 onto one or moreimages. For example, as shown in FIG. 37, various output images areillustrated which may be provided to a user for review on one or moredisplays disclosed herein (e.g., for review in an augmented realityenvironment and/or otherwise). As shown, the output images includeoriginal input image 3702 (e.g., captured during step 3515 as previouslydiscussed), original input image 3704 (e.g., captured during step 3530as previously discussed), output image 3734, and output image 3736.

As shown, output image 3734 corresponds to input image 3704 with acontour 3705 corresponding to colorimetric change 3703 projectedthereon. Output image 3736 corresponds to input image 3702 with contour3705 also projected thereon.

In this regard, it will be appreciated that the processing performed byregistration module 3710 permits corresponding locations in any of theinput images 3702, 3704, and/or 3706 to be cross-referenced andidentified as desired. In addition, the processing performed bysegmentation module 3720 permits the colorimetric change 3703 to beidentified. As a result, the location of the colorimetric change 3703may be identified in any of the input images, and particularly in inputimages 3702 and 3706 where the colorimetric change 3703 is not otherwisevisible. As shown, the processing circuitry may project contour 3705corresponding to colorimetric change 3703 on input images 3704 and 3706to provide output images 3734 and 3736, respectively. In otherembodiments, a segment of input image 3704 corresponding to colorimetricchange 3703 (e.g., identified by segmentation module 3720 during step3550) may be projected on input image 3702 and/or 3706 to illustrate thecolorimetric change 3703.

In step 3560, the processing circuitry provides the projectedrepresentation (e.g. contour 3705 and/or colorimetric change 3703)superimposed on one or more images to a user, robot, and/or othermachine in the form of one or more images. For example, a user may viewthe projected representation in an augmented reality environment. Inthis regard, the user may use the projected representation of thecolorimetric change to identify areas of potential contamination on thesurface of interest 3701, even after the ADS is removed (e.g., in step3535).

Accordingly, in step 3565, a user, robot, and/or other machine mayperform decontamination operations on the surface of interest 3701 usingthe projected representation without requiring the ADS colorimetricchange to still be present. As a result, decontamination may be moreconveniently performed and with high accuracy even after the ADS isremoved from the surface of interest 3701.

Embodiments of the invention have been described in language more orless specific as to structural and methodical features. It is to beunderstood, however, that the entire invention is not limited to thespecific features and/or embodiments shown and/or described, since thedisclosed embodiments comprise forms of putting the invention intoeffect. The invention is, therefore, claimed in any of its forms ormodifications within the proper scope of the appended claimsappropriately interpreted in accordance with the doctrine ofequivalents.

What is claimed is:
 1. A method comprising: receiving a first image of asurface exhibiting a colorimetric response; receiving a second image ofthe surface without the colorimetric response; processing the firstimage by a segmentation module comprising an artificial neural network(ANN) to identify the colorimetric response; and projecting arepresentation of the colorimetric response identified by thesegmentation module on the second image to permit a user to identify alocation of the colorimetric response on the surface after thecolorimetric response has been removed from the surface.
 2. The methodof claim 1, further comprising processing the first image and the secondimage by an image registration module to align the representation on thesecond image.
 3. The method of claim 1, wherein the representationcomprises a segmented portion of the first image.
 4. The method of claim1, wherein the representation is a contour corresponding to a locationof the colorimetric response on the surface.
 5. The method of claim 1,wherein the projecting comprises providing the representation to a userin an augmented reality environment.
 6. The method of claim 1, furthercomprising providing to the ANN a first plurality of training imagescomprising the colorimetric response and a second plurality of trainingimages without the colorimetric response.
 7. The method of claim 1,further comprising preprocessing the first image and the second imagebefore the processing to perform one or more of: adjusting a size,applying a filter, detecting edges, adjusting a color space, adjusting acontrast, and/or adjusting a brightness.
 8. The method of claim 1,wherein the colorimetric response is associated with an interactionbetween an interrogating material applied to the surface and a threatmaterial disposed on the surface.
 9. The method of claim 8, wherein theinterrogating material is an Agent Disclosure Spray (ADS) and the threatmaterial is a chemical, biological, radiological, nuclear, and/orexplosive material.
 10. The method of claim 1, wherein the surface is asurface of interest associated with a vehicle or a person.
 11. A systemcomprising: an image acquisition device configured to capture a firstimage of a surface exhibiting a colorimetric response and a second imageof the surface without the colorimetric response; and processingcircuitry configured to: process the first image by a segmentationmodule comprising an artificial neural network (ANN) to identify thecolorimetric response, and project a representation of the colorimetricresponse identified by the segmentation module on the second image topermit a user to identify a location of the colorimetric response on thesurface after the colorimetric response has been removed from thesurface.
 12. The system of claim 11, wherein the processing circuitry isconfigured to process the first image and the second image by an imageregistration module to align the representation on the second image. 13.The system of claim 11, wherein the representation comprises a segmentedportion of the first image.
 14. The system of claim 11, wherein therepresentation is a contour corresponding to a location of thecolorimetric response on the surface.
 15. The system of claim 11,wherein the processing circuitry is configured to provide therepresentation to a user in an augmented reality environment.
 16. Thesystem of claim 11, wherein the processing circuitry is configured toreceive at the ANN a first plurality of training images comprising thecolorimetric response and a second plurality of training images withoutthe colorimetric response.
 17. The system of claim 11, wherein theprocessing circuitry is configured to preprocess the first image and thesecond image to perform one or more of: adjusting a size, applying afilter, detecting edges, adjusting a color space, adjusting a contrast,and/or adjusting a brightness.
 18. The system of claim 11, wherein thecolorimetric response is associated with an interaction between aninterrogating material applied to the surface and a threat materialdisposed on the surface.
 19. The system of claim 18, wherein theinterrogating material is an Agent Disclosure Spray (ADS) and the threatmaterial is a chemical, biological, radiological, nuclear, and/orexplosive material.
 20. The system of claim 11, wherein the surface is asurface of interest associated with a vehicle or a person.